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Introduction 


Signed into law in 2013, the Local Control Funding Formula (LCFF) ushered in a new 
system of accountability and support that encourages a deeper and more meaningful approach 
to the work of educational improvement in California. Specifically, LCFF gives school districts 
more flexibility in how they invest resources to meet locally defined goals. Under this new 
system, districts are required to detail their plans for improving student outcomes and how 
dollars will be spent, with particular attention to performance problems highlighted on the 
state’s School Dashboard,1 which emphasizes the achievement of vulnerable student 
populations. 


Under emerging policy structures in California, the responsibility for school 
improvement is increasingly placed upon local school districts, with County Offices of Education 
(COEs) playing a critical support role. In this system, districts are responsible for school 
improvement, with counties in charge of ensuring quality across districts and providing 
feedback and support where necessary. Underlying this major policy shift is the idea that local 
leaders are in the best position to drive real educational improvement and ensure quality 
across multiple schools and contexts. 


Through such bodies as the State Superintendent’s Accountability and Continuous 
Improvement Task Force,2 policymakers have signaled that “continuous improvement” is the 
recommended path to achieving better outcomes for California’s students. While a continuous 
improvement approach involves many changes to the way educational systems work, strategic 
data use is central to effective implementation (Hough, Willis, et al., 2017; Grunow, Hough, 
Park, Willis, & Krausen, 2018). Although there is wide agreement that data are critical to the 
state’s policy vision, many questions remain: What does it look like to use data for 
improvement in California’s context? What data are needed, by whom, and for what purpose, 
at various levels of the education system? And what can we learn from innovative practice in 
our state? 


As California supports districts and counties statewide to embark on this improvement 
journey, there are important lessons to be learned from the CORE districts, six of which 
developed an innovative accountability system under a waiver from No Child Left Behind 
(NCLB). The CORE districts are early adopters of the new accountability paradigm: local leaders 
using multiple measures of school performance and working together to figure out collectively 
what works best for struggling schools. Now deepening their work together as a Networked 
Improvement Community (NIC),3 the CORE districts have simultaneously expanded access to 
their multiple-measures data and learning system by inviting other California districts to join 
their “Data Collaborative.” Districts who join contribute their own student data to the CORE 
measurement system and are then able to benchmark student performance against other 


1 For more detail on the California School Dashboard, see: https://www.caschooldashboard.org/#/Home. 
2 The Task Force report can be found at https://www.cde.ca.gov/ta/ac/ar/documents/account-report-2016.pdf. 
3 See Bryk, Gomez, Grunow, and LeMahieu (2015) for more information about NICs. 
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schools in the state. Data Collaborative districts engage in structured network learning activities 
that enhance their capacity to use data to drive system improvements. Currently, over 50 
California school districts representing nearly a million students have joined the Data 
Collaborative. With the one million students already served by the original CORE districts, the 
combined data system now represents nearly a third of California’s student population. 


In what follows in this report, we first provide a framework for how data use for 
improvement is different from data use for accountability and how data should be used by 
actors at different levels of the system. Next, we discuss the policy context in California and the 
current state of data use based on interviews conducted in the summer of 2017 with 41 leaders 
from state education agencies, COEs, school districts, technical assistance providers, education 
advocacy organizations, and education associations. Finally, we share lessons from the CORE 
Data Collaborative about how data can be used for improvement within networked structures, 
including what data is needed and how learning and collaboration can be facilitated. 


What Is Data Use for Improvement, and How Should It Be Used? 


Though the use of data in the education system is not a novel practice, data use for 
continuous improvement reflects a recent and critical change in mindset. Data use for 
improvement entails a cycle of collecting and interpreting data, constructing ideas on potential 
solutions, making appropriate modifications to improve, and monitoring and researching 
whether changes resulted in improvement (Abbott, 2008; Easton, 2009; Hamilton et al., 2009; 
Ikemoto & Marsh, 2007; Mandinach, 2012; Means, Gallagher, & SRI International, 2010). In its 
basic form, this iterative cycle transforms data into usable knowledge and thus makes it 
actionable. The term continuous improvement also signifies the collaborative, transparent, and 
inquiry-driven culture that sets it apart from other approaches, such as accountability. In this 
section, we review extant literature and provide a framework for how to use data for 
improvement within the educational context. We will cover three primary topics: 


1. Data use for improvement is different from data use for accountability purposes. 

Data must be embedded in an aligned process for improvement. 

3. The kinds of data needed vary by user, by improvement process phase, and by 
educational system levels. 


ee 


Data Use for Continuous Improvement Is Different from Accountability Metrics 


A primary focus of data use in the field of education in recent history has been on 
system-level measures for accountability. For many years under NCLB, educational 
measurement was limited to standardized assessments of student academic performance, 
measured annually for accountability purposes. The Every Student Succeeds Act (ESSA), passed 
in 2015, has introduced a more comprehensive approach to measurement than was required 
under NCLB, with the intention of including more measures and moving away from adverse 
consequences of NCLB’s measurement system, namely the narrowing of the curriculum 
towards tested subjects and content, strategic gaming of accountability structures, and 
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cheating (Figlio & Getzler, 2002; Jacob & Levitt, 2003; Lauen & Gaddis, 2016; Neal & 
Schanzenbach, 2010). Specifically, ESSA requires states to include multiple measures of student 
academic achievement, including: academic performance as measured by proficiency on 
English Language Arts and math tests; academic growth; graduation rate; development of 
English Learner (EL) proficiency; and at least one additional indicator of “School Quality or 
Student Success.” Acceptable additional indicators include measures of student engagement, 
educator engagement, student access to and completion of advanced coursework, post- 
secondary readiness, and school climate and safety. While more measures certainly provide a 
nuanced view of school or district performance, additional measures alone will not result in 
continuous improvement (American Association of School Administrators, 2002; Darling- 
Hammond, Cardichon, & Melnick, 2017; Farrell, 2015; Jimenez & Sargrad, 2017). To this end, it 
is critical to distinguish how data for accountability is different from data for improvement. 


We start here with a framework modified from the healthcare context for how data can 
be used to advance system change. As Figure 1 illustrates, there are two established pathways 
through which quality measurement is linked to quality improvement: the “selection” pathway 
and the “change” pathway. In a “complete improvement strategy,” data is used for both 
selection and change, but these data sources and their use are different (Berwick, James, & 
Coye, 2003). The first pathway, selection, is what we typically think of as accountability. Data 
utilized for selection help parents decide where to send their children and guide policymakers 
in determining whether particular schools are in need of intervention due to low performance 
(as in the case of identifying the bottom 5 percent of schools under both NCLB and ESSA). Such 
consumers utilize knowledge about performance to make “selections” about interventions, 
choice of schools, or policies, but the selection pathway does not use data to understand, make 
changes to, or improve the underlying system. 


The second pathway, however, stresses the importance of using data to signal the 
improvements needed to those responsible for enacting change in the system. Data for this 
pathway are employed to evaluate changes, run predictive analytics, and establish priorities 
(Yeager, Bryk, Muhich, Hausman, & Morales, 2013). Within education, data for improvement 
can help track whether schools are taking action to meet their goals and enhancing teachers’ 
capacity to adapt their instruction to their particular students (Breiter & Light, 2006; 
Schildkamp & Kuiper, 2010; Schildkamp & Lai, 2013; Schildkamp, Poortman, Luyten, & Ebbeler, 
2017; Wayman, 2006; Wohlstetter, Datnow, & Park, 2008; Young, 2006). Broadly, data for 
change treat knowledge-seeking as crucial to managing the school system, as opposed to 
independent of the system, and prioritize an adaptive and continuous design for immediate 
improvement (Baily, Bottrell, Lynn, & Jennings, 2006). The primary users of this kind of data are 
the people responsible for running the systems, including school, district, and county leaders. 
These local leaders utilize data about processes and results to make changes to the educational 
systems that they are in charge of managing.4 


4 There is yet another important use for education data, which is the use for research, a critical activity for 
knowledge generation. While investigation for improvement purposes prioritizes an adaptive and continuous 
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Figure 1. Two Pathways to Quality Improvement in Education 
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Figure 1. Adapted from "Connections between quality measurement and improvement," 
by Berwick et al. (2003). Medical Care, 41(1), |-30. 


The annual public data provided for accountability might illuminate a problem in 
performance, but the data in the change pathway help to highlight why a problem is occurring 
and how change can be enacted to make improvements. The types of data needed for 
improvement are not purely quantitative but rather should encompass what W. Edwards 
Deming referred to as “profound knowledge,” or the “knowledge of systems,... of variation,... of 
psychology, and... of how knowledge grows” (Lewis, 2015). To this end, data for improvement 
are collected over time, are intermediary in nature, and are not solely represented by the “use 
of percentiles... and [measures that yield] nearly all Os or all 100%” (Yeager, Bryk, Muhich, 
Hausman, & Morales, 2013; Provost & Murray, 2011). With schools and districts drawing 
knowledge from a widening range of data that includes input (e.g., student demographic data), 
process (e.g., classroom observations), context (e.g., building information), and output data 
(e.g., student achievement), a crucial step moving forward is to determine what types of 
information would most benefit educators in this quest for continuous improvement (Nelson, 
Slavit, & Deuel, 2012; Schildkamp et al., 2017). 


Before looking into the specifics of data use for improvement in education, however, it 
is important to realize that the change pathway, unlike selection, relies “on the ability of 
organizations and providers of care to undertake systematic change” (Berwick et al., 2003). In 
other words, there is no reason to believe that data will result in improvement if the 


design for immediate improvement, investigation for research purposes broadly aims to “contribute to 
generalizable knowledge” (Baily et al., 2006). However, many recent research approaches, such as research- 
practice partnerships (Coburn, Penuel, & Geil, 2013) or improvement research partnerships (Bryk, 2015), 
specifically aim to leverage research for local improvement. In the ideal state, these research partnerships are 
designed to support education leaders at the various stages of improvement detailed in Figure 2. The statewide 
data systems for which Phillips, Reber, and Rothstein (2018) advocate in their work for this project could be useful 
for both generalizable research and for specific types of improvement work. 
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measurement system is not developed and used in the context of an aligned improvement 
process. We turn to this idea next. 


Data Must Be Embedded in an Aligned Improvement Process 


At the core of continuous improvement is the perspective that data do not drive. In 
other words, a distinguishing feature of data use for improvement is that data do not guarantee 
use or action; individuals must be active in all steps of data collection and interpretation to 
make them actionable (Ackoff, 1989; Farrell, 2015; Mandinach, 2012; Marsh, Pane, & Hamilton, 
2006). To this end, the optimal way to utilize data for improvement is in the context of an 
aligned continuous improvement methodology.s Various continuous improvement 
methodologies are currently used in education, each articulating a set of tools, principles, and 
social practices. These include the Plan-Do-Study-Act Cycle (PDSA), Network Improvement 
Communities, Design-Based Implementation Research (DBIR), Deliverology, Six Sigma, Lean, 
Implementation Science, Positive Deviance, and Appreciative Inquiry (Park, Hironaka, Carver, & 
Nordstrum, 2013). Regardless of the specific methodology selected, experts agree that it is 
important for an organization to select and use acommon method to promote discipline within 
their continuous improvement efforts. Having a common improvement methodology creates a 
shared language and enables building expertise with the practical tools of improvement over 
time. When an organization is aligned around this improvement methodology, the data that are 
collected can be used in a clear and actionable manner. Across all methodologies, several 
features of continuous improvement remain constant: 


Cycles of inquiry. One central feature of continuous improvement is the idea of cycles of 
action and reflection. This cyclical process allows for a “design-development ethic,” which 
prioritizes constant learning at a low cost by making improvements in increments (Yeager et al., 
2013). An interesting example of this kind of cycle in education comes from the College 
Readiness Indicator Systems (CRIS), which is a new framework for measuring college readiness 
created by Brown University, Stanford University, and the University of Chicago (Annenberg 
Institute for School Reform, Brown University, John W. Gardner Center for Youth and Their 
Communities, Stanford University, & University of Chicago Consortium on Chicago School 
Research, 2014). The CRIS “cycle of inquiry” consists of identifying, planning, implementing 
supports, monitoring progress, adjusting supports, and analyzing results. The key to constant 
learning is for districts and schools to collect frequent data implementing changes in order to 
effectively monitor progress and adjust supports. Importantly, the people who are tracking the 
data in these cycles should also be responsible for making good use of it. 


Norms, culture, and mindset. To make good use of data, educators and administrators 
must act as the agents of change in an improvement process. Frontline participants in an 
organization should exercise actively the practice of reflection, discourse, and action followed 
by the consistent review and monitoring of progress (Adelman, 1993). To this end, a critical 
feature of continuous improvement is the establishment of norms and culture surrounding data 


5 For more on improvement methodologies in education, see Hough, Willis, et al. (2017). 
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use that encourage action from individuals and teams. In the context of education, scholars 
define data culture as an environment within schools and districts that “includes attitudes, 
values, goals, norms of behavior and practices” along with a shared vision for data use among 
teachers and leaders (Mandinach, 2012). Additionally, effective data use for improvement 
requires buy-in, as educators are more likely to employ data to inform their decision-making if 
they believe in the use of data and the importance of the specific data elements being tracked 
(Mingchu, 2008; Schildkamp et al., 2017). This necessitates a mindset that data help to 
illuminate and solve problems, rather than to scapegoat or punish. Agents of change must 
perceive data use as a path to “improve” rather than “prove;” an improving stance entails using 
information as a means to better understand their students and systems and to make changes 
to their practices accordingly (Nelson et al., 2012). A proving stance towards data use, on the 
other hand, views information as an end rather than a means, and is demonstrated when 
educators use data to show how well students are performing rather than to highlight areas of 
improvement or rethink practices based on data results. 


Organizational capacity. A supportive data culture involves both collaborative inquiry 
among teachers, fostered by school and district leaders, as well as an organizational structure 
and methodology that promotes data use for improvement (Datnow, Park, & Wohlstetter, 
2007; Hough et al., 2017; Grunow et al., 2018; Kerr, Marsh, Ikemoto, Darilek, & Barney, 2006; 
Sharkey & Murnane, 2006; Wayman, Cho, & Johnston, 2007; Wayman & Stringfield, 2006; 
Wohlstetter, Datnow, & Park, 2008; Young, 2006). The capacity to make collaborative inquiries 
and create change out of information ultimately “depends on the cognition of the individuals 
operating within [schools] (e.g., teachers and administrators)” (Moss & Piety, 2007). Agents of 
change must have the skills to understand, interpret, analyze, and utilize data to recognize their 
school and student needs and adjust their practices accordingly (Mandinach, 2012; Marsh, 
Pane, & Hamilton, 2006; Means, Gallagher, & SRI International, 2010; Schildkamp, Poortman, 
Luyten, & Ebbeler, 2017; Sharkey & Murnane, 2006; Supovitz & Klein, 2003; Wohlstetter, 
Datnow, & Park, 2008; Young, 2006). Certain routines, such as professional learning 
communities in which educators work with and learn regularly from each other, can help 
support capacity-building around effective data use for improvement (Datnow, Park & 
Wohlstetter, 2007). Equipping “frontline workers” with a broader set of tools and measures for 
recognizing “the signals for improvement initiatives” can also support educators in building 
capacity (Martin, Nelson, Rakover, & Chase, 2016). Providing these tools and a structure for 
their use at the school or district level can help support and advance organizations’ use of data 
for improvement. 


Specific Data Needed Vary by User and Phase of the Improvement Process 


While an aligned improvement process with cycles of inquiry, an appropriate culture 
and mindset, and organizational capacity opens the door for effective data use for continuous 
improvement, the specific types of data necessary to make changes and improvements 
ultimately depend on who is using the data and on the phase of the improvement process. In 
order to determine what data use for improvement looks like in our education system in 
California, it is helpful to first draw from a broader structure for improvement (Provost & 
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Murray, 2011). Figure 2 illustrates a typical improvement process with five basic phases, 
starting with the project definition phase and ending with the phase where system managers 
are sustaining and monitoring improvements that have been made (The Victorian Quality 
Council, 2008). The data needed at each of these phases are quite different. In the project 
definition phase, data needs are geared toward understanding the problem. During the 
diagnostic phase, managers evaluate existing processes within that area of interest to diagnose 
potential quality problems and opportunities for improvement. These activities are often 
geared towards helping leaders “see the system,” or better understand the systemic causes of 
poor performance (Nayfack, Park, Hough, & Willis, 2017). During the intervention phase, 
individuals use data within iterative PDSA cycles for implementation and progress monitoring. 
At the impact measurement phase, data help evaluate the effect of interventions on 
predetermined performance measures. And finally, in the sustainability phase, data are used 
for monitoring and refining interventions as well as providing feedback in order to sustain the 
improvements. While some of the information that these phases require is readily available, 
much of the quantitative and qualitative data need to be collected throughout the process, 
often with much greater frequency than typical data collection and reporting timelines. 


Figure 2. Overview of Improvement Process 


Organizational governance framework 
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Figure 2. Adapted from The Victorian Quality Council (2008). P = Plan, D 
= Do, S = Study, A = Act 


Within education, the data used in this improvement process will vary by level of the 
educational system and the actors within each level. These system levels consist of classrooms, 
schools, districts, as well as counties and state-level actors. The data used for improvement at 
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each of these levels are different, based on the kinds of information that individuals need in 
order to make changes in their sphere of influence. An example of these different kinds of data 
can be found in Figure 3, which provides sample indicators for measuring college readiness at 
each system level (derived from the CRIS). While classroom-level indicators track individual 
student-level data that assist educators in recognizing the students in need of additional 
support, school- and system-level indicators are utilized to evaluate whether the right 
environment is provided for promoting college readiness (Annenberg Institute for School 
Reform et al., 2014). Specifically, school-level indicators trace school supports available to 
students, and system-level indicators assess the use of district and broader system resources to 
further college preparedness. In what follows here, we dig deeper into data use at each of 
these levels as well as the alignment among them. 


Figure 3. Sample Menu of CRIS Indicators and Supports 


= CLASSROOM-LEVELINDICATORS —_{@) SCHOOL-LEVEL INDICATORS @) SYSTEM-LEVEL INDICATORS 


@ ACADEMIC GPA and credits/courses G Advanced Placement course availability Student/teacher assignment policies 
PREPAREDNESS |G Benchmark exams © Academic supports Number of schools with Advanced 

& Consistent grading standards Placement courses 
Availability/evaluation of academic 
supports 


ACADEMIC No/low disciplinary infractions Students’ perceptions (instructional Communicated expectations about 
TENACITY Attendance scaffolding, academic press, support for academic tenacity 

autonomy) Professional development on practices 
Professional development on practices that promote academic tenacity 

that promote academic tenacity 


Self-discipline 
Mastery goal orientation 


COLLEGE & Completion of college and financial aid & College-going culture in school G Resources to support college-going 
KNOWLEDGE applications G Access to counseling resources culture/knowledge 
G Campus visits & Resources for teachers’ college & Communicated expectations about 
G Meetings with college advisor knowledge college knowledge supports 


Figure 3. Adapted from Annenberg Institute for School Reform et al. (2014). 


Classroom level. Individual- or student-level data indicators focus on tracking students’ 
individual progress and learning to guide teacher practice (Provost & Murray, 2011). Data at 
this level, according to the Teaching for Robust Understanding Framework, should allow 
teachers to help their students across five primary dimensions: content, cognitive demand, 
equitable access to content, agency, authority, and identity, as well as formative assessment 
(Schoenfeld, 2015; Schoenfeld & The Teaching for Robust Understanding Project, 2016). Given 
these dimensions, student-level data would ideally include disaggregated student data to track 
variation by student, attendance records, and grades as well as portfolios for each student to 
target interventions. For instance, a data support tool used in New York City Department of 
Education schools in 2001 supplied disaggregated data reports in the beginning of the school 
year as well as access to an online information system with individual student data available to 
educators (Breiter & Light, 2006). The reports, which were connected to teaching resources 
with strategies and activities for standards-based learning, assisted teachers in using the 
provided data points to target their instruction, cater to diverse learners, and facilitate 
conversations about individual students with other teachers and parents. 


School level. At the school level, data indicators assess the quality and allocation of 
current resources and supports for students (Wohlstetter et al., 2008). This data can help a 
school leadership team track progress over time and look for variation between classrooms or 
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grades to determine how to allocate resources and supports. Examples of school-level data 
would be: longitudinal student data by classroom, grade, measure, and subgroup; information 
about special programs; and professional development opportunities. Prince George’s County 
Public Schools in Maryland provide an example of a powerful use of school-level data through 
their early warning system (Parton, 2015). The district created the early warning system 
through analyzing previous student data to predict what were effective indicators of academic 
success in ninth grade. Schools within the district established their own plans for change based 
on grade-level meetings in which teachers reviewed comprehensive student data. Through 
these plans for action that were updated quarterly, schools in Prince George’s County Public 
Schools aimed to identify and allocate academic support to the subgroup of students in danger 
of not graduating on time. 


System level. At the system 
level, referring to the district and 
beyond, data are used to look at A groundbreaking data-sharing initiative called the Silicon 


patterns in opportunity allocations and Valley Regional Data Trust provides an opportunity for 
to inform funding infrastructure and districts within Santa Clara, Santa Cruz, and San Mateo 
. : counties to share real-time information about their students 
policy (Annenberg Institute for School 
an amongst themselves and among health and human service 
Reform et al., 2014). Data indicators for county agencies. The shared data warehouse, which the 


districts, for example, would include Santa Clara County Office of Education houses, allows 
administrative data and longitudinal districts, schools, and teachers to view various metrics from 
student data by grade, subgroup multiple real-time databases, such as student information 
eae : systems and state and locally determined assessments. 
measure, and building that would ! ae 
: ; an While the warehouse supplies the counties with a 
illuminate variations by school and comprehensive and comparative picture of the system 
identify “bright spots” in performance levels within it, teachers, schools, districts, and even partner 
or policy implementation. District agencies can also pull from the “dashboards” the 
leaders can use this data to make aggregated or disaggregated data indicators that they need 

was ‘ to make changes at their respective system levels. 
decisions about which schools need e e Y 

; : : (McNulty, 2017) 

support and on what dimensions in 
order to ensure quality district-wide. 
Counties play a similar role, but would attend to variation across all of the districts supported 
by the COE. Finally, this kind of system-level data can help the state identify and solve large- 
scale policy issues, such as teacher recruitment and retention or school funding. 


A County-Level “Bright Spot” 


Integration across levels. While specific data indicators will vary across the classroom, 
school, and system levels to meet the needs of various users, alignment across the levels is 
critical (Coburn & Talbert, 2006; Farrell, 2015; Kerr, Marsh, Ikemoto, Darilek, & Barney, 2006; 
Schoenfeld, 2015; Supovitz, 2006; Wayman, Jimerson, & Cho, 2012). These system levels should 
not operate within silos; rather, a “strong bottom-up information flow from school-level 
participants” is essential to informing improvement system-wide (Wohlstetter et al., 2008). 
Data from the classroom and school levels should thus ultimately flow to the district, county, 
and state levels on the “data highway” (Mandinach, 2012). This alignment in data sources is 
supported by alignment in improvement priorities. For example, teachers’ efforts in the 
classroom should be supported by the improvement work at the district level, with each actor 
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understanding how the data they are tracking is related to a broader improvement strategy. 
This clear communication will allow for better alignment of efforts from the ground up and 
from the top down. In this way, classroom- and school-level improvement work should 
illuminate system-level barriers that must be addressed by the district, county, or state. 
Similarly, district-, county-, or state-level goals should focus improvement efforts or approaches 
at the school sites and within classrooms. This alignment can be driven by system-level actors 
acting as the central hubs of data (Moore, Bracco, & Nodine, 2017), although more study is 
needed on the best way for these systems to be developed, maintained, and used at different 
levels (Mandinach, 2012; Marsh & Farrell, 2015; Means et al., 2010; Reeves & Burt, 2006). In 
light of this framework, we now turn to a discussion of the statewide context around data in 
this new policy era and the extent to which California’s current structures support data use for 
improvement. 


California Policy Context 


In California, recent policy shifts have created a high degree of local control with the 
expectation that school districts will think differently about school and district improvement. 
Specifically, several new state policy structures—LCFF, the California School Dashboard, and the 
Local Control Accountability Plan (LCAP) process—have given school districts more autonomy in 
how they allocate resources, how to evaluate student and school performance, and how they 
choose to intervene when groups of students are not achieving (Humphrey & Koppich, 2014). 
Under these new approaches, district leaders create goals in each of the state’s eight priority 
areas, and then allocate resources strategically to ensure equitable outcomes for all students.« 
Central to this new accountability system is a commitment to supporting and cultivating a 
system-wide culture of “continuous improvement.” 


Although K-12 education stakeholders in California have become familiar with terms 
such as “continuous improvement” over the past several decades, these words exist in a 
different context under the state’s new policy structures. While not specifically outlined or 
defined in the LCFF statute, bodies such as the State Superintendent’s Accountability and 
Continuous Improvement Task Force and the State Board of Education have issued reports, 
memoranda, and statements of model practice signaling that continuous improvement is the 
recommended path to achieving better outcomes for California’s students. The State 
Superintendent’s Task Force on Accountability and Continuous Improvement’s report 
(California Department of Education, 2016) explains the state’s vision for continuous 
improvement: 


We are equally committed to supporting and cultivating a system-wide culture of 
continuous improvement... that relies on multiple measures to provide the information 


6 California’s eight priority areas are (a) basic services, (b) implementation of state standards, (c) course access, (d) 
student achievement, (e) other student outcomes, (f) student engagement, (g) parent involvement, and (h) school 
climate. For more information on California’s eight priority areas, see 
https://www.cde.ca.gov/Fg/aa/Ic/statepriorityresources.asp. 
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necessary to engage in improvement efforts. This culture of continuous improvement 

must infuse all parts of the system, including continuous improvement for the system 

itself, through evaluative mechanisms that allow us to learn from local experience and 
revisit the indicators, tools, and systems of support we use to ensure they are working 
as intended. 


The ability of districts and COEs to engage in continuous improvement hinges on their 
capacity to use data for improvement and to enact change at the district office and in schools 
across the system. The state’s California School Dashboard, launched in March 2017, offers a 
publicly available, online set of data incorporating a range of state and local measures aligned 
to support and inform school improvement efforts.7 Districts are required to use data from the 
state’s School Dashboard to inform the development of their LCAP. 


The California School Dashboard broadens the concept of accountability from a single 
metric—standardized test scores—to one with multiple indicators. The indicators include six 
state measures (academic, English learner progress, chronic absenteeism, graduation rate, 
suspension rate, college/career readiness) and four local measures (basic services, 
implementation of state academic standards, parent engagement, and school climate). COEs 
have two additional local indicators: coordination of services for expelled youth and 
coordination of services for foster youth. 


The five performance levels of the state’s School Dashboard are color-coded—red, 
orange, yellow, green, blue—with red signifying the lowest performance level and blue the 
highest. In contrast to the state’s previous measure of academic performance, the Academic 
Performance Index (API), which assigned a single number between 200 and 1,000, performance 
levels on the School Dashboard are calculated based on how current performance compares to 
past performance and whether those results have declined or improved. The state goal is to 
reach green on all state indicators, and a performance level below green indicates a need for 
improvement. By design, the School Dashboard does not provide statewide averages for any of 
the indicators or an overall ranking of a school or district. The School Dashboard’s home page 
emphasizes the state’s priority of addressing achievement gaps with an “Equity Report” that 
highlights a school’s or district’s low-performing students from among a dozen subgroups. The 
goal is that this data will encourage improvement at various levels of the system, which 
includes meaningful stakeholder engagement.s 


The data contained in the California School Dashboard is derived from the state’s 
student data system, the California Longitudinal Pupil Achievement Data System (CALPADS). In 
2012, after nearly a decade of work, the California Department of Education (CDE) completed 
this K-12 student database, which includes individual student data on school attendance, 
special program participation (e.g., English Learner and special education programs), course 


7 To learn more about the Dashboard’s model of accountability, see https://www.cde.ca.gov/ta/ac/cm/. 
s For more detail on California’s accountability system and the California School Dashboard, see Polikoff, Korn, and 
McFall (2018). 
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enrollments, and state test scores. Each school district in the state is expected to report these 
data to CDE, which compiles them in a statewide data system following students across time 
(Warren & Hough, 2013; Phillips et al., 2018). In addition to the California School Dashboard, 
these data can be accessed through pulling reports from the Education Data Partnerships and 
DataQuest.10 Additionally, local test coordinators can access student assessment data from the 
California Assessment of Student Performance and Progress (CAASPP) portal.11 


The State of Data Use in California 


To evaluate the extent to which California’s school districts are engaging in continuous 
improvement using these new policy and support structures, we interviewed 41 leaders from 
state education agencies, COEs, school districts, technical assistance providers, education 
advocacy organizations, and education associations.12 Our aim in these interviews was to 
understand how various actors define continuous improvement, to what extent they are 
engaged in it, and how state structures, especially the School Dashboard, are helping or 
hindering these efforts. We analyzed each case (i.e., agency, office, or organization) individually 
to determine the nature of engagement in continuous improvement and data use, and then we 
systematically analyzed patterns and factors associated with patterns across cases (Miles & 
Huberman, 1994). We also reviewed state policy materials including State Board of Education 
reports, memos, and statements of model practice. In this section, we discuss the three main 
ideas we derived from the interviews: 


1. The California School Dashboard provides needed information about school 
performance but is insufficient for continuous improvement. 

2. To better support continuous improvement, school districts are developing local data 
systems, but there is wide variation in capacity to do so across the state. 

3. The capacity to use data for improvement remains challenging for most districts. 


California’s School Dashboard and continuous improvement. In our interviews, the 
School Dashboard garnered mixed reactions. For some districts and COEs, the School 
Dashboard offers a baseline of data and increased opportunities to review and analyze their 


9 https://www.ed-data.org/ 

10 https://www.cde.ca.gov/ds/sd/cb/dataquest.asp 

11 http://www.caaspp.org/ 

12 Interviews were conducted during the summer of 2017 and utilized semi-structured interview protocols, which 
were recorded and transcribed. Our research team conducted 41 interviews with 56 leaders from state agencies 
(n=10), county offices of education (n=3), local education agencies (n=20), education associations/advocacy 
organizations (n=3) and technical assistance providers (n=4). Our purposeful sample consisted of technical 
assistance providers and state level leaders who were responsible for executing continuous improvement 
strategies. County offices of education and local education agencies were similarly selected based on identified 
engagement in continuous improvement, and also to represent variation in enrollment, geography, urbanicity and 
student demographics. To analyze, we used a multiple case study design (Yin, 2013), conducting cross-case 
analyses (Miles, Huberman, & Saldafia, 2013) to identify common definitions and practices across respondent 
groups (e.g., district leaders, support providers, and policymakers). 
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data, and it puts a spotlight on student populations that are underperforming. As one state 
policy maker characterized it, the School Dashboard provides a “standardized tool to start the 
conversation [for districts].” This is particularly the case for smaller districts with limited 
research capacity. One county administrator reported that, for one such district, the School 
Dashboard data revealed problems of practice that were previously invisible: 


Just having the School Dashboard to point out that, “Hey, you have the lowest indicator 
overall for math.” That one superintendent was very surprised by that, and when we 
talked a bit more with him, he had not yet adopted materials for the Common Core 
math. We now have a team out there working with him—it was because of the 
conversations around the data. 


Another county administrator pointed to the impact of the School Dashboard on 
highlighting vulnerable student populations: 


We provided a training for districts to ask, “What are the data saying about foster youth, 
[English learners] and low-income [students]? And what kind of actions make a 
difference?” Wanted them to think about it in terms of their LCAP. They were 


overwhelmed by it. We are coming back to this conversation this year because they now 
have data on this. 


In these instances, data from the School Dashboard help to illuminate problems in 
performance at the system or school level. However, respondents brought up two main issues 
with using the School Dashboard data for improvement. 


First, many district and COE leaders were concerned that the state is working with 
“post-mortem data” and that this limits their ability to engage in continuous improvement. 


State assessment data are not recent, leading one district leader to describe using state data as 
an “autopsy”: 


The state data is great and it helps give us a good picture of what happened the year 
before, but it’s really an autopsy. We need to be able to look at the data as the year 
progresses and get that data into the hands of our sites and our teachers so that they 
can make adjustments accordingly. 


A county office administrator also conveyed concern about the lag in data and its impact 
on districts to engage in continuous improvement strategies: 


[Districts are concerned about the delay in release of the state’s School Dashboard. If 
the School Dashboard’s not going to come out until December this year, and that’s 
already halfway through the year.... what do we do from July until December, when it 
comes to looking at strategic planning? 
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Second, the data available on the School Dashboard is currently limited in that it offers 
only a static, summative view of whether or not progress has been made from year to year. For 
this reason, the majority of respondents noted the importance of more frequent data to 
monitor progress and change course when needed. One county office administrator offered 
this example: 


When it comes to continuous improvement, it’s a recognition that we are expected to 
respond frequently and immediately... so that we can respond to data, and shift the 
system in order to meet the need of the students on a more regular, more frequent 
basis. 


Having more frequent data is critical to informing the various phases of the 
improvement process, particularly within PDSA cycles. For this reason, districts and COEs are 
increasingly turning towards local data systems that enable the ongoing, systematic collection 
and analysis of real-time local data. These data systems help educators at all levels of the 
system to identify and develop more specific, targeted supports and to make immediate 
adjustments throughout the school year to improve educational outcomes for students. We 
turn to this idea next. 


Local data systems for continuous improvement. More than half of the 20 district 
leaders interviewed for this study reported conducting systematic analyses of locally generated 
data to inform their improvement efforts. Some district leaders are developing increasingly 
robust internal data systems that include multiple measures and access to local, real-time data 
such as student attendance, grades, or assessments. Other districts are also incorporating 
stakeholder survey data (e.g., school climate surveys) and observational data garnered through 
classroom and school site “walkthroughs.” Approximately 11 of the districts interviewed had 
already developed and implemented internal data systems, and three were in the process of 
developing systems. As one administrator described: 


We have a local dashboard system that we report on our own, local data a couple of 
times a year. Then, we also have a data warehouse, which is a live connection to our 
student information system and our student achievement records that everybody from 
teachers, to principals, to counselors, to assistants can use to look on a daily basis to 
things like student attendance or student grades or student achievement. 


A district superintendent shared how their district has similarly been developing a data 
warehouse with more formative day-to-day data that are accessible to everyone throughout 
the school year: 


For example, instead of waiting for suspension data, we want to get referral data in 
there this year so we can track that school culture from day to day. Are we seeing an 
increase in referrals? Are we seeing referrals from certain teachers or parts of the 
school? And address those issues more frequently. 


14 | Using Data for Improvement: Learning from the CORE Data Collaborative 


However, many respondents asserted it would be helpful for district and school leaders 
to have systems that would enable them to see associations between changes in strategies and 
changes in outcomes at various levels of the system. However, current data systems are not 
adequate for helping districts monitor progress against a specific goal or target for a specific 
improvement project. As one technical assistance provider stated: 


If you’ve got an improvement project trying to find gaps, and your employees need two 
or three strategies to get there, and you want to monitor both your implementation of 
the strategies right next to what’s happening with chronic absence in the targeted 
classrooms or schools, there’s no system that lets you do that, that | know of. Which 
means that it requires the local user, maybe whoever is leading that project locally, to 
sort of cobble this stuff all together. 


COEs also caution that while these efforts are promising, districts are in different phases 
of having their own data systems and have different levels of organizational capacity. Some 
smaller districts, in particular, do not have research departments to implement or manage the 
types of internal data systems that support improvement efforts. Noting the wide variation 
among districts, one COE administrator said: 


Our districts are in different phases of having their own data systems where they can 
produce more timely information that can be used to inform instruction and adjust 
practices during the year. It just depends on what the district has and can pull forward.... 
I’m not sure that the districts have gotten to a place where they can make really strong 
use of that data to inform what they do. 


Building capacity to use data for improvement. Because “data don’t drive,” simply 
having more frequent data, while necessary for continuous improvement, is insufficient. People 
at all levels of the system also need to know how to make good use of data—analyzing it to 
understand variations in performance and evaluating whether new investments have changed 
outcomes. As discussed above, this requires strong organizational capacity to support skill 
development among practitioners along with the norms, culture, and mindsets to support data 
use. A district administrator described the need this way: 


| think one of the pieces is that we need to be better at helping our site administrators 
know how to lead [data] conversations. And how to help them to be very versed in the 
data. Not only what the data says and where they see the gaps, but what they can do to 
change that outcome. 


We found not only a wide variation in district capacity to use data for improvement but 
also examples of promising practice and concerted effort around building capacity. For 
example, a number of the district leaders we interviewed described having district-wide and 
school-level teams who conduct regular, systematic reviews of data to inform instructional 
change and to determine what additional support systems and mechanisms are needed to 
promote student learning and achievement. These teams can include board members, cabinet 
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members, central office administrators, school site leaders, teachers, and classified staff. 
Several district leaders noted that they situated this work initially at the district level, before 
delving in more deeply at the school site level. For other district leaders, this process is 
embedded at the school level in Professional Learning Communities (PLCs) and Instructional 
Leadership Teams (ILTs). A district superintendent described ILTs as the “unit of 
implementation” at the school site level: 


We’ve invested a lot in building our site leadership capacity and instituting an 
Instructional Leadership Team at each of our sites.... To varying degrees of quality, 
they’re engaged in analysis of data, of reflection on needs, and then putting in plans to 
address those needs. 


Another superintendent pointed to the role of PLCs in analyzing data for improvement: “So at a 
teacher level, in a PLC, they are looking at very real-time data associated with their instruction, 
and they’re able to make those kinds of changes regarding lesson planning and lesson delivery, 
on amore real-time approach.” 


District leaders are also working to create a culture in which data analysis and 
monitoring are ongoing as well as to increase the capacity of district personnel at all levels to 
collaborate around using data for improvement. One district superintendent described it this 
way: “What | mainly focus on is building leadership capacity around mindsets that are 
necessary to get fear out of the organization.... Creating safety’s been very important. Creating 
competency at the site level was important....” Another district superintendent noted: 


| think that we’ve successfully shifted the culture around viewing improvement as not a 
thing to be afraid of and shifting this notion around punitive accountability to we all are 
accountable for improvement.... Now | think it’s common practice for teachers to get in 
each other’s classrooms. It’s common practice for principal meetings to be focused on 
student data and instruction rather than compliance. It’s common practice that we’re 
able to and willing to look at the brutal facts of where we are with a lot of our kids and 
resist making excuses or blaming kids or families because of that, because of those 
results. 


In a continuous improvement framework, people at all levels of the system need to 
know how to make good use of data, how to analyze it to understand disparities in 
performance, and how to evaluate whether new investments have changed outcomes. This 
requires skill development among practitioners and often, as noted above, a shift in the culture 
of data use. And sometimes, as one district administrator observed, this can also simply mean 
unlearning poor practices and prior assumptions about data use: “I think probably one of the 
biggest barriers is to help people overcome just un-useful practices.” 


Districts also need support to learn how to more effectively use data to inform 


investments and to document when an investment is strategic. In many districts, administrative 
functions are siloed, and there is a lack of understanding about program evaluation or how to 
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align goals, programs, and dollars. A district administrator described the complexities of the 
new system and the challenges it presents for districts: 


Because we now have these School Dashboard indicators and we have three years of 
state assessment data now, it’s figuring out are the dollars that we have allocated to 
these actions and services, are they working? If they’re not, should we reallocate funds 
and focus that in other areas so that we can really get the improvement that we’re 
looking for? It’s not as black and white as it was years ago. 


State policymakers, technical assistance providers, and COE administrators interviewed 
similarly identified the need to increase districts’ capacity and expertise to generate and utilize 
more timely local data to impact the classroom. One state policymaker acknowledged, “There 
needs to be a conversation on how we invest in resources to empower districts to use the data 
they collect and not wait for a School Dashboard. There’s definitely a hole in our system.” 


Learning From the CORE Data Collaborative 


To fill this gap, over 50 school districts have joined the CORE Data Collaborative for 
access to better data and a collaborative structure for its use. The Data Collaborative is an 
outgrowth of the original CORE districts’ work and is designed to provide better data for 
educators as well as an infrastructure for building capacity to make use of data for 
improvement. The CORE Data Collaborative provides a unique opportunity to understand and 
learn from districts and counties that have opted into a data system to supplement the state’s 
data. 


To understand the data and analytics available in the CORE measurement system and 
how these resources are being used across the state, we interviewed leaders from nine districts 
and two COEs that are members of the CORE Data Collaborative and interviewed staff in the 
“CORE hub,” the intermediary organization that supports the work of participating districts. We 
also observed in-person and virtual Data Collaborative meetings and reviewed training and 
meeting materials. In addition, we reviewed preliminary findings from interviews with 16 
district and county CORE Data Collaborative members conducted by McKinsey & Company.13 In 
what follows, we will first provide background on the CORE districts and the development of 
the CORE Data Collaborative. Then we will discuss the data and analytics available in CORE’s 
measurement system and the capacity-building and networking opportunities available to 
members. Finally, we will share the benefits and limitations to the Data Collaborative in 
participants’ own words. 


Background on CORE and the Data Collaborative 


The CORE districts first developed their measurement system under their NCLB waiver. 
The guiding principle of the CORE districts in their use of multiple measures under this 


13 Our interviews were conducted in Summer 2017; McKinsey & Company interviews took place in December 2016. 
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accountability framework is that data should be used as “a flashlight, not a hammer.” 14 In other 
words, the CORE districts emphasized that measures should be useful for continuous 
improvement and not just for accountability. At the same time, they believed that measures 
should represent all of the aspects of school and student performance, not just the academic 
subjects that had been emphasized under NCLB. As a result, CORE’s measurement system from 
its inception has focused on academic outcomes alongside non-academic measures of student 
success, including chronic absenteeism, suspension/expulsion, students’ social-emotional skills, 
and school climate and culture.15 While ESSA and LCFF now prioritize multiple measures as well, 
CORE was ahead of this policy shift with its systematic measurement of school and student 
performance on multiple dimensions, and generated widespread interest in the popular press 
and policy circles nationwide (Blad, 2015; Bornstein, 2015; Zernike, 2016). 


Since the passage of ESSA in 2015, the CORE districts have reconceptualized their 
partnership, as they were no longer required to implement their shared accountability system. 
Through developing and executing their waiver plan, the CORE districts established a deep 
commitment to collaboration and shared learning around their multiple-measures data system. 
For this reason, in 2016, CORE reorganized as a Networked Improvement Community 
committed to: 


1) Anchoring their shared work in the student and school quality measures in their 
measurement system, and holding each other accountable for specific outcomes; 

2) Looking inward and building coherence within each district as a way to improve the 
power of each system to improve student outcomes (Fullan & Quinn, 2015); and 

3) Using the tools of improvement science to anchor their collaborative work together 
as a Networked Improvement Community (Bryk, Gomez, Grunow, & LeMahieu, 
2015).16 


While deepening their work together as a NIC, the CORE districts simultaneously 
expanded access to their multiple-measures data and learning system by inviting other 
California districts to join their Data Collaborative. Any district was invited to join for a relatively 
small fee, which includes access to the data system and shared learning opportunities around 
its use.17 Districts that join contribute their own student data to the CORE measurement 
system, all of which is currently in their administrative datasets and sent to the state annually. 
Participating districts are then able to benchmark student performance against other schools in 
the data system. Data Collaborative districts engage in structured network learning activities 
that enhance their capacity to use data to drive system improvement. As of the 2017-18 school 
year, over 50 California school districts representing nearly a million additional students are 
participating in the Data Collaborative, including San Bernardino City Unified School District, a 
consortium of districts in the Silicon Valley (the East Side Alliance coordinated by the Silicon 


14 http://coredistricts.org/calif-core-districts-pilot-accountability-measures/ 

1s For more detail on CORE’s implementation of the waiver, see Marsh et al. (2016). 

16 For more detail on CORE’s early implementation as a NIC, see Nayfack et al. (2017). 

17 At the time of this report, the fee was $28,000 per partnering agency plus $0.28 per student for professional 
learning support. This, for example, results in a fee of $30,800/year for a district with 10,000 students. 
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Valley Education Foundation), the Sweetwater Union High School District, Aspire Public Schools, 
Green Dot Public Schools, and school districts in Sacramento, Ventura, and Riverside counties. 
When combined with the one million students already served by the original CORE districts, the 
integrated data system now represents nearly one third of California’s student population. 


CORE’s Data System 


The majority of the data used in CORE’s measurement system is contained in the state’s 
student data system (CALPADS). CORE gets the data directly from the districts, but all of the 
data is processed on an annual basis. However, CORE makes this data system more valuable for 
district leaders by (a) adding additional data sources, (b) using this data to calculate different 
kinds of metrics, and (c) creating analytics and data displays that enable district and county 
leaders to look at data in different ways. These data displays are made available to the districts 
more quickly than the same source data on the California School Dashboard or other public 
data sources.19 


Figure 4 displays the data elements in CORE’s measurement system. Many of the data 
elements are also represented in California’s School Dashboard and other state data reporting 
tools, including academic achievement as measured by the Smarter Balanced Assessment 
Consortium (SBAC) in English Language Arts and mathematics, graduation rate, chronic 
absenteeism, and suspension/expulsion. However, the CORE measurement system includes 
additional data sources, which extend the state’s data in important ways. First, participating 
districts can choose to administer surveys of students’ social-emotional learning (self- 
management, growth mindset, self-efficacy, and social awareness) and student, parent, and 
staff reports of a school’s culture and climate (support for academic learning, sense of 
belonging and school connectedness, knowledge and perceived fairness of discipline rules and 
norms, and sense of safety).20 Additionally, in an effort to continue improving and extending 
the data system, college outcomes have recently been added to the CORE data system via the 
National Student Clearinghouse.21 For the 2017-18 school year, college attendance rates by 
school are included in the data system, with additional metrics being developed for future 
years. 


1s See http://coredistricts.org/our-work/data-collaborative/. 

19 In the past two years, participating districts have seen results on the CORE dashboard by mid-August, whereas 
California state data are not available until December. 

20 For more detail on how CORE’s surveys can be used for accountability and continuous improvement, see Hough, 
Kalogrides, and Loeb (2017). 

21 http://www.studentclearinghouse.org/ 
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Figure 4. Data Elements in CORE’s Measurement System 
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Additional items in the CORE measurement system represent methods for utilizing 
CALPADS data to produce more informative metrics for use by educators. For example, CORE 
has developed and reports an academic growth metric, which is very different from how 
California state reports test scores. The CORE measure of student growth takes into account an 
individual student’s prior test history, socioeconomic disadvantage, disability status, English 
Learner status, homelessness, and foster care status, using this information to measure how 
quickly they grow relative to students similar to them in these categories. The CORE model also 
accounts for concentration of these characteristics within schools. In this way, the CORE growth 
measure is constructed as a “value added” model, estimating the school’s impact on student 
achievement relative to other schools serving similar students. Alternately, California’s 
accountability system uses one-year school-wide change over time in student achievement, 
which does not account for changes in student population or the kinds of students a school 
serves. Additionally, CORE reports a high school readiness score at the middle school level, 
which is based on predictive analytics demonstrating that combinations of student GPA, 
attendance, grades, and suspensions are indicative of a student’s risk level upon entering high 
school.22 Finally, CORE calculates EL redesignation rates in a way that reflects district leaders’ 
concerns about long-term ELs and stagnant redesignation within schools. At the school level, 
this is calculated as the percentage of students who are reclassified from English Language 
learner status to “fluent English proficient” out of the number of all the English learners who 


22 A student is considered ready for high school if they: (a) have an eighth-grade GPA of 2.5 or higher; (b) have 
attendance of 96% or higher; (c) have no grades of D or Fin ELA or math in the final course grade; and (d) were not 
suspended in eighth grade. At the school level, this is reported as the percentage of students who are high school 
ready upon completing eighth grade. 
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are reclassified at a school site in the current year plus all those English learners who, after five 
years, were not reclassified at that school.23 


However, arguably the most valuable aspect of the CORE data system is the variety of 
ways that data can be displayed. All participating districts have access to an interactive data 
interface in addition to a more static report card. The static report, which was created as part of 
the waiver, compares schools by level (elementary, middle, high) on a 1-10 scale for each 
metric, and breaks out each metric by student demographics (race/ethnicity, EL, FRPL, SPED).24 
In our past work in the CORE districts, we found that educators appreciated that these reports 
contained multiple measures, anchored their school data compared to others, and showed how 
school performance on each metric had changed over time (see Marsh et. al, 2016). 


As useful as these static reports can be, as discussed in the framework above, they have 
limited use for improvement. They do not help to narrow attention or explain the nature of the 
problem. This is where the interactive data interface can be important; it was constructed to 
enable simple, deeper analysis that enables the user to dig into the kind of information that can 
be found in a static report. 


Specifically, the CORE interactive data interface allows for a variety of comparisons 
between and within schools. First, the tool provides scatterplots to compare any two variables 
and how schools are distributed across them. As shown in Figure 5, this can involve comparing 
two outcome variables such as academic achievement and growth in mathematics. A 
comparison such as this can show which schools have both high academic achievement and 
higher performance scores (in the top quartile); these sites might be “bright spots,” or places to 
learn from promising practices. On the other hand, schools with both low performance and low 
growth may need serious intervention to improve. A user can also look at achievement against 
demographics to see how schools of different sizes or poverty concentrations are performing. 
To enable this kind of comparison, there are a number of filters that can limit comparisons to 
similar schools (by size or demographics) or subgroups. The tool can also focus on schools 
within a given region, for example, within a particular district or county. A comparison like this 
allows for a school leader to ask, “How does my school compare to others like mine?” A district 
leader can ask, “How are the schools in my district doing, and how does this compare to other 
districts?” While a county leader can ask, “How are my districts, and the schools within them, 
performing?” Each dot on the scatterplot represents a school that is identified by name to 
facilitate support and collaboration. 


23 CORE’s measure of EL proficiency is slightly different than what is specified in ESSA. Rather than using only test 
score results to determine progress on English proficiency, the CORE districts chose to report reclassification rates, 
which are a combination of language proficiency scores and academic performance (Carranza, 2015). 

24 Fresno and Long Beach Unified School districts have made these reports public; they can be accessed at 


reports.coredistricts.org. 
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Figure 5. Scatterplot of Two Variables on the CORE Interactive Data Interface 


Comparing Growth in Math to Proficiency in Math, All Students 
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Most of these analyses compare two variables, which is a very useful way of 
understanding relationships. However, the CORE data system has many measures, and our 
previous research has demonstrated that a school’s performance can vary widely across 
indicators (Hough, Penner, & Witte, 2016). CORE’s interactive data interface provides a bird’s- 
eye view of how all of the schools in a district or region are performing on a given indicator 
compared to the others. As an example, Figure 6 shows each indicator compared to chronic 
absence. Through California’s five-level color coding, a user can see the proportion of schools 
that are in each category within a system (e.g., district or county). Each histogram counts the 
number of schools in each performance category, conditional on their performance category in 
the chosen metric. This view allows a user to see patterns across all of the indicators. For 
example, when comparing student reports of school culture-climate to chronic absence, it is 
apparent that there is a relationship between culture-climate and absenteeism: More schools 
that are red on chronic absence are also red on school culture-climate, and on the other hand, 
schools that are blue on chronic absence are more often doing well on culture-climate. This 
view allows users to see that while schools with red chronic absence often have red culture- 
climate, this is not necessarily true for the relationship between academic growth and culture- 
climate. As this bird’s-eye view illustrates, CORE’s interactive data interface provides ways for 
the user to drill down into more detail, for example, to see which particular schools are low on 
each indicator and to highlight patterns. 
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Figure 6. Bird’s-Eye View of All Performance Metrics 
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CORE’s interactive data interface also allows for further investigations to understand the 
problems within a particular school. In these kinds of comparisons, the user can see how 
performance on a particular metric has changed over time and how this compares to the 
district and overall trends. For example, Figure 7 shows that in a particular school the students’ 
ratings of school culture and climate are lower in the target school (at only 61 percent of 
students reporting a positive school culture-climate in 2016-17) than in the district or Data 
Collaborative overall.2s Benchmarking in this way can be useful, especially on a new measure 
like school culture-climate, which is required by the state but may lack meaning without being 
normalized against other schools. In this way, a school or district can know whether a score of 
61 is “good” or “bad” relative to other schools. A user can also look at reports for specific 
subgroups as a way to get more information. 


25 CORE’s culture-climate measure includes four constructs: support for academic learning, sense of belonging and 
school connectedness, knowledge and perceived fairness of discipline rules and norms, and safety. To create 
school-level metrics, CORE translates survey responses into the percentage of positive responses in each school. 
For example, a school with a student culture-climate score of “80” would indicate that 80 percent of the survey 
questions were answered positively by students. For more information, see Hough, Kalogrides, et al. (2017). 
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Figure 7. Trends Over Time in a Particular School Compared to District and Overall Trends 
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The CORE interactive data interface also allows for directly analyzing performance gaps 
between different groups of students or grades within a school, as shown in Figure 8. For this 
school, what looks like average performance (a student growth percentile of 59) is revealed to 
be a serious problem in sixth grade, where students are growing at a much slower rate than the 
rest of the network, while students in seventh and eighth grade demonstrate high growth. A 
result such as this helps to direct attention—instead of solving what could be thought of as a 
school-wide problem, district and school leadership can focus on supporting the sixth-grade 
team. 
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Figure 8. Breaking Down School-Level Results by Grade 
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Analyses such as this one are supported by the drill down function within the data 
interface, which allows the data and research leads in each district to get a results extract, with 
student-level detail, of the records that make up the scores for each school. Districts can use 
the extract for their own research or analytics about trends and patterns, or they can take this 
data to a school team to discuss the performance of particular students. If districts have their 
own capacity to pull data from their student information system, then this extract may be less 
useful for creating a research file. However, as discussed above, for districts with limited 
research or data capacity, it can be very helpful to have all the metrics in one extract, making it 
much easier than merging many files together from multiple data files. 


Data Use in the CORE Data Collaborative 


The majority of districts interviewed joined the Data Collaborative at the invitation of 
their COEs, and most signed up during the 2016-17 school year. COEs joined in order to have 
access to a larger data set and to get better data on district performance across different 
groups—data which would not otherwise be available to them. Participating districts similarly 
expressed their interest in being part of a larger data set and viewing their data through a 
different lens. As one district administrator explained, “They’ve got a really good data system. 
We could get feedback on our scores in a different way from what the California Department of 
Education is giving us.” COEs report that they are using the CORE data system to help them 
track the performance of individual districts and schools. In this way, being part of the CORE 
data system provides data that is not readily available through the state. As one COE 
administrator noted, “The idea of seeing how districts or seeing our school sites and grades 
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within a school site are faring over time is something that CORE can provide that the current 
data set doesn’t provide at the state level.” Participation in the Data Collaborative also provides 
districts with a way of utilizing data that is not currently available to them unless they have 
robust research offices in the central office. District and COE leaders reported using the data in 
several different ways. 


First, users report employing the data to compare their school(s) to others in the 
collaborative. This ability to use data to enable cross-district comparisons is regarded by many 
district and county leaders as one of the main value-added features of the CORE data system. 
One district administrator shared: 


[Comparing our schools to others] was particularly eye opening because we have a few 
of our Title | schools that we’re very concerned about the score, but when you compare 
them... they’re doing better than expected. They’re still low, but better than expected. 
Gives us a different viewpoint on it. 


This kind of comparison can also help to highlight problematic outcomes that might not be 
visible when only looking at one site’s data. As one district administrator recounted: 


When we started working with CORE and we were able to see the reclassification [rates] 
of other large urban districts at an elementary level, we were blown out of the water 
because they were reclassifying at a much higher rate. When those kids were 
reclassified out of elementary they were set up for success in middle school and high 
school.... We can see a pattern based on the availability of that data, how it was 
affecting their scores as the kids moved up... 


The administrator subsequently shared these findings with the district’s director of English 
language programs who immediately started contacting some of these districts to learn about 
their programs and practices to support English learners. 


Second, respondents reported finding particular value in the unique indicators in the 
data system. CORE’s student growth model is viewed by Data Collaborative members as an 
especially important and powerful tool and as one that is particularly valued by district leaders, 
as this administrator noted: 


The CORE growth metric is the single most important piece of data we get from CORE 
because we can use it to help high-performing schools realize that high scores may not 
be good enough if their students aren’t growing at the same rate as similar students in 
other school districts. It’s also a powerful tool for communicating to a low-performing 
school that has made significant gains with [its] students in comparison to similar 
students in other districts. It can help all principals determine how well they’re doing in 
comparison to how well they “should be” doing.... A district research department could 
do half of that, and determine individual student growth from year to year, but the 
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CORE data provides us perspective because it answers the other half of the question— 
“Is that good?” —by giving us the comparative data to similar students across California. 


District leaders also found value in the student surveys, even if they do not collect the 
data themselves. The analysis conducted with these data help district and COE leaders 
recognize broad patterns in the data that can be applied to their own sites. For example, one 
district superintendent described seeing important patterns in the student survey data 
available through CORE: 


... |Saw a [CORE] data set of some districts who did [the social-emotional survey] with 
kids [and] they noticed a big fall off in the female students around self-efficacy... and so 
it just makes you question, “What are female students in sixth, seventh and eighth grade 
going through?” | think that this data set really sparks a lot of conversation... 


Finally, as described above, the CORE data set allows users to drill down to student level 
outcomes, and Data Collaborative members found this tool to be valuable. Data leads can 
access the student level records that make up the scores for each school on CORE’s interactive 
data interface, or as one district superintendent characterized it: “This data set allows you to 
look at the kids.” 
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Capacity-Building and Collaboration to Support Data Use 


Respondents reported 
learning a lot from the CORE 
data system, which helps them 
better define their problems of 
performance within schools and 
districts. However, these data by 
themselves do little to help 
district or school administrators 
determine why a problem is 
occurring or to identify what 
changes might be enacted to 
make improvements. This is 
where CORE’s capacity-building 
and collaborative structures 
come into play. CORE believes 
that these opportunities for 
networked capacity-building and 
support around data use are 
another way in which they differ 
from what is more generally 
provided by the state: 


By putting a program 
lead together with a data 
lead, and demonstrating 
how those folks can work 


Example of Monthly Session Offered to Data Collaborative Members 


A recent monthly session’s theme was “Using the displays to better 
understand chronic absence for a school in your district.” CORE staff first 
walked participants step-by-step through where to find data related to 
chronic absence in the various CORE data sets, and then through a series 
of “quick explorations” to learn how these data can help to “illuminate 
the attendance story” for their schools and districts. Some of the 
available measures and analyses for chronic absenteeism the group 
reviewed included: 


Chronic absence percentages and performance levels 

Time series graphs by student group and overall 

Bar charts for student groups over time 

Time series comparison to other districts 

Scatterplots to show chronic absence plotted against SD, ELA 
scores, math scores and suspension rates 


Participants were then asked to consider questions such as: 
What patterns do you see between schools in your district? 
What variability exists between “like” schools in your district? 
Is chronic absence increasing or decreasing over time? 
How do the gaps differ by student group? 


These explorations and reflections were then followed by a presentation 
on the chronic absence initiatives being implemented by a leader in one 
of the member districts, who presented concrete practices that they have 
used to address chronic absence in their schools, and linked these efforts 
to data. 


together, and how the different information can work together to actually engage ina 
way of addressing a particular issue, is | think one of the things that we do that’s 
different than the state’s doing... that’s one element, and maybe the most crucial 


element that we engage in... 


Network activities include monthly virtual meetings that cover a range of topics such as 
“trends in the CAASPP results,” “high school readiness,” and “English Learner measures.” In 
addition, there are biannual in-person meetings that are attended by five-person teams from 


each partnering educational agency. 


These in-person meetings offer opportunities for members 


to network, share learning, and better understand how to analyze performance and outcome 
data. All of these meetings are designed to provide a hands-on, interactive opportunity for 
members to build their capacity to utilize performance and outcome data, and to share learning 
with each other. As a CORE leader described it: 


| think what we’re doing is we’re equipping our reps, our data leads who are there, with 
the ability to use the data that’s here, to do some analytical exploration into one of the 


28 | Using Data for Improvement: Learning from the CORE Data Collaborative 


areas represented by the data. To look for variability, to look for best practice 
possibilities. To look for schools that maybe have gaps or needs that aren’t surfacing in 
any other way, or in any other data set. Then a light frame for how one might engage in 
learning more about what’s going on at that particular school. 


To this end, CORE focuses on providing Data Collaborative members with training on 
how to engage in an improvement process: 


This year we’re really focusing on the improvement science frame we have. It’s around 
the idea of “seeing the system,” doing a causal system analysis, [using] the tools— 
process mapping and fishbone diagrams, and empathy interviews—as opportunities to 
try to get more than simply the hard data.... How do | start from the [data] and then 
actually drill into something that’s an area of interest and try to get more information 
about what this looks like for my particular school? 


This idea for how to support Data Collaborative members flows from the work of the 
CORE Improvement Community (CIC). As we detailed in a recent report (Nayfack et al., 2017), 
the CIC is still in the beginning stages of its continuous improvement work, but the districts 
collectively have learned that accessing and interpreting different types of data are critical to 
building a complete understanding of a problem of practice. For the CIC, analysis began by 
examining a set of quantitative data reports provided by the network hub. Based on data 
showing clear gaps in performance, the districts identified math achievement for African 
American and Latinx students as their target, and then collected additional data to help them 
think more critically about the next set of activities that would help deepen their team’s 
understanding of the problem of practice. This included interviewing teachers and students and 
observing classrooms to clarify some of the trends that surfaced in the system-wide data. These 
qualitative data, which are clearly not part of any shared data system, were a pivotal part of 
understanding how math was being taught and why math instruction and achievement 
disparities persisted. As one district improvement lead explained: 


We have a lot of great data we use on a regular basis, but some of the things we need to 
measure we don’t have a way of measuring unless we go and talk with the kids and hear 
their stories. That’s a complicated and time-consuming process, but it might be the most 
effective to figure out how to solve the problems we are encountering. 


This process of learning about the system causes of problems highlighted by data in the 
shared data system is reflected in the activities offered to members of the CORE Data 
Collaborative. In all of the activities, it is emphasized that quantitative data and analytics from 
the CORE data system are an important aspect of understanding a problem but are only the 
beginning of the improvement process. 


Many of the members we interviewed had only recently joined the Data Collaborative 


and thus had limited participation in network meetings, yet they still found them useful and 
appreciated the network of training and support. We observed some of these meetings, and 
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several members in attendance expressed how much they valued the opportunity to network 
with other members and to establish ongoing relationships. One district participant described 
how much he learns from seeing other districts’ approach to using data, noting that, 
“Sometimes another district is looking at data in a different way, putting a twist on it.” 


Limitations of the CORE Data Collaborative 


Based on our interviews and observations, we concluded that while members were 
generally positive about the network and the support provided by CORE, they were in different 
stages of understanding and utilizing the CORE data set. Participants have expressed a need for 
expanded onboarding support, and for training targeted to stakeholder groups (e.g., principals, 
board members). More specifically, members recommended bi-weekly webinars, some to help 
new districts’ onboarding and others targeted at different ways to use the data. They also 
recommended on-demand, short e-learning videos for each type of data that districts might 
analyze. These recommendations point to the need for various users to deeply understand data 
before they are able to act on it. 


Beyond training and support, some members suggested that CORE could play the role of 
matchmaker, identifying and connecting lower performing districts with those that are showing 
strong improvement, as described by one district administrator: 


There are probably 30 or so districts now in CORE. But, more than me, at the district 
level, if I’m a principal, it would be great if they actually gave Principal Smith... a list of, 
“Here are three other schools in our CORE Collaborative that are similar to you that 
have similar kids and similar backgrounds, that are having bigger improvements. So, 
here are those three schools, and here are the phone numbers for those three 
principals.... We’d love to get you guys together to talk.” Or, even better, would be like, 
“We'd like to bring you together for a session.” You know, that session could be 
electronic, or it could be in person, but if they actually used the data and got down to 
identifying best-practice principals by different kinds of subjects and groups and they 
actually made those connections, they did that matchmaking, | think that’d be pretty 
exciting. 


Several district leaders also expressed the desire to be able to identify “what steps, 
programs, services created the results” and to identify leading indicators. One district 
administrator likened it to “reverse engineering,” while another described the process as 
“backwards mapping”: 


| think what would be helpful would be sort of being able to map back. Okay, so if we 
were looking at something like A through G completion... going to map it backwards to 
where is the earliest possible place you could start to identify where you’ re having 
problems in that area, and that you could do something about? 
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Members agree that while being able to identify children in danger of failing sooner 
rather than later is critical, the more important question is what to do about it? There is a 
strong desire for the CORE Data Collaborative to help identify and disseminate information on 
what kinds of programs have been effective at changing course. They also noted a desire for the 
data provided by the CORE Data Collaborative to be more “user-friendly and actionable.” 


Finally, CORE recognizes that its data system has some limitations when it comes to 
supporting continuous improvement, as one CORE team participant acknowledged: 


The [data system] that we have is essentially an annual dashboard... It’s helpful for 
doing your annual school site planning. It’s helpful to see after the fact, so it is post- 
mortem how the school is moving over time or changing over time.... It’s valuable and 
meaningful in a continuous improvement framework because it’s a part of... the work 
that schools and districts can do to engage in “seeing the system.” 


Data Collaborative members concur that while this summative data is useful for 
strategic planning or studying trends, it is not immediately actionable for improving student 
achievement, particularly in the intervention phase of an improvement process, when they are 
involved in iterative testing. To move beyond using data to “see the system” and identify gaps 
or patterns in the data, districts and schools must then turn to locally generated data in order 
to fully engage in the process of continuous improvement. This kind of local data, specifically on 
frequent data on student learning, is essential to changing instructional practice at scale. The 
CORE data appears to be most useful for district and county leaders, and even for school 
administrators or leadership teams, but potentially less useful for classroom teachers. 


Conclusion and Policy Implications 


Data use for continuous improvement, with its adaptive and iterative nature, differs 
from data use for other purposes. Most importantly, data used in an improvement process 
must support the needs of the user: Classroom teachers need data to support their students, 
school leaders need data to support all of the students and teachers in a school, and district 
and county leaders need data that helps them decide where to provide additional supports or 
resources as they work to ensure quality across sites. These uses are very different from the 
data needed at the state level to hold districts and schools accountable for educational 
performance. 


For data to be utilized effectively for improvement, there must be a deliberate structure 
around their use, with various actors aligned in both goals and process. The specific data 
necessary to drive improvement also depend on the phase of the improvement process. For 
example, at the beginning of an improvement process, data needs are geared towards 
understanding the problem and its causes. When educators are introducing changes, however, 
data are used in real-time to test whether those changes are resulting in expected 
improvements. While some of the information that these phases require is readily available, 
much of the quantitative and qualitative data needs to be collected throughout the process, 
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often with greater frequency than typical data collection and reporting timelines currently 
taking place within California’s districts. 


While the California School Dashboard has been welcomed as an improvement over 
previous accountability measurement systems, such as the Academic Performance Index, 
policymakers and educators across the state agree that California’s existing data systems are 
limited as a resource for districts to review, analyze, and make decisions to improve the system. 
Specifically, the state data system suffers from the following limitations: (a) The static nature of 
the displays makes it difficult to study trends over time; (b) the “one school at a time” displays 
make it difficult to compare across or between systems, to benchmark performance, or to 
identify “bright spots;” (c) limited breadth of data and lack of comparisons across metrics make 
it difficult to build a holistic, integrated understanding of performance for a school; and (d) the 
inadequate frequency of data collection limits the ability of schools to track real-time impact as 
they make changes to improve. 


To support improvement, CORE’s data system provides educators in the CORE Data 
Collaborative with tools and supports that make the data more useful. Currently, school 
districts representing over one third of California’s student population have joined the CORE 
Data Collaborative, which is designed to provide better data for educators and an infrastructure 
for building capacity to make use of data for improvement. Districts that join contribute their 
own student data to the CORE measurement system, all of which is currently in the districts’ 
administrative datasets and is sent to the state annually. CORE makes this data system more 
valuable for district leaders by (a) adding additional data sources; (b) using this data to calculate 
different kinds of metrics; and (c) creating analytics and data displays that enable district and 
county leaders to look at data in various ways. Participating districts are then able to 
benchmark student performance against other schools in the data system and are invited to 
participate in structured network learning activities that enhance their ability to use data to 
drive continuous system improvement. 


Current participants in the CORE Data Collaborative find this data to be particularly 
useful for them in the “project definition phase” of an improvement project, when they are 
analyzing data to build a deeper understanding of the problem. However, for other phases of 
an improvement process—particularly when testing interventions—these annual data are less 
useful. When districts reach this phase of an improvement cycle, there is a need to collect and 
track different kinds of data. This includes data that are locally generated, tracked, and used at 
the classroom level and school level as well as more frequent data than is currently available in 
any shared data systems. Given the widespread demand for better data and capacity-building 
to support improvement, the state could provide assistance in these areas. To this end, we 
provide three directions that the state could take to better support local efforts around data 
use for improvement: 


1) Ensure that all districts have access to better tools to make use of data already 
collected by CDE. As discussed above, CALPADS data are the backbone of the California School 


Dashboard and other state data systems, but these source data are not made available to 
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districts or those who support them in ways that are easy to use. CORE makes this data system 
more valuable and user-friendly by creating better data displays, calculating innovative metrics, 
and adding new data sources, such as surveys and links to higher education. This data system is 
now available to any district that is able to pay a small fee to join the Data Collaborative. Now 
that the infrastructure has been developed by CORE for California data,26 the additional cost for 
each district to join is relatively low and could be supported by the state. For example, the state 
could provide funds for districts to join a data system like the CORE Data Collaborative, which 
could incentivize better local data use. CORE’s development of this data system has been 
supported by philanthropic investment and can now be leveraged as a statewide benefit. This 
approach of leveraging the development of technologies and systems is a strategy that the 
state in general could employ to spread innovation. Alternately, California could develop and 
manage a state data system, as many other states in the nation have done successfully.27 


2) Facilitate the development of local and real-time data systems to support 
improvement. District and county leaders we interviewed discussed unanimously the 
importance of local data for monitoring progress and evaluating the impact of interventions. 
Such data include intermediate outcomes, such as student attendance or results on formative 
assessments, which are often collected in student information systems but may need to be 
extracted and formatted in order to be tracked over time and used by actors across system 
levels. The most useful data systems for improvement will also include process and input data 
that help leaders track implementation of particular interventions. These types of data are not 
typically collected at scale in school systems, such as the amount of exposure students are 
getting to an intervention (e.g., tutoring or pull-out support), results from student surveys, data 
from observations on classroom instruction or other school-based activities, measures of 
quality of implementation, or finance data to connect expenditures to programs. As discussed 
earlier, the specific data elements tracked in an improvement process must be locally 
generated and aligned with the particular improvement initiatives underway within schools and 
districts. However, the state could help strengthen the infrastructure for these local data 
systems and their use in two ways. 


First, the state could enable local data use by investing in data structures that could be 
utilized by districts or those that support them. There are a number of local and real-time data 
systems currently being utilized across the country, as detailed in the Appendix, including many 
within California districts. Given that California districts have many similarities in terms of 
context and data that are already collected, the state could invest in developing one or more of 
these platforms for use by districts across the state. The most promising route would be for 
California to join the Ed-Fi standards, given Ed-Fi’s growing national prominence. Ed-Fi is an 
open-source technology that consists of a uniform data standard, a centralized operational data 
store (ODS) and application program interface (API), and various mapping tools for visualizing 


26 CORE’s data warehouse and interactive data interface are managed by Education Analytics 
(http://edanalytics.org), which supports states and districts around the country in using data to make informed 
decisions on policies and programs. 

27 See Phillips et al. (2018) for examples of how other states have been successful in building data systems, 
including addressing concerns around data security and privacy. 
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data. Ed-Fi’s uniform “Ed-Fi Data Standard” allows districts using Ed-Fi to connect student 
information, assessment, and rostering systems, as well as to develop and share new 
applications for using locally generated data. California’s investment in Ed-Fi technology could 
support counties or consortia of districts, like CORE, in accelerating data use for improvement 
and cost-sharing across districts. 


Second, the state could continue to work with assessment providers to ensure that the 
formative assessment system is optimally designed for alignment and integration with local 
data systems. Interim data on assessments are an ideal source of formative data for schools, 
teachers, districts, and consortia, but it can be difficult for educators to access and make use of 
the data. For example, if the SBAC interim assessments being administered across the state 
were aligned and compiled in a central location, it would be easier to use the interim data in 
local data systems, to share across sites, and to build statewide capacity around their use. 


3) Ensure access to high-quality professional learning opportunities around using data 
for improvement. As discussed above, “data don’t drive” and development of data systems 
alone will not result in improvement without a strong capacity to make use of data at all levels 
of the system. To this end, we recommend that the state play a role in building the capacity of 
actors across the system to use data for improvement. While this support could come from a 
number of agencies across the state, including the CDE, the California Collaborative for 
Educational Excellence, the California County Superintendents Educational Services Association, 
or other consortia like CORE, there must be alignment in approaches to ensure that various 
professional learning activities are coordinated and support a shared vision of continuous 
improvement for the state. Key in this regard is the development of common improvement 
frameworks to guide district and school leaders in their analysis, exploration, idea generation, 
implementation, and tracking efforts. Additionally, the state could support a process for 
utilizing data to identify (a) where improvement is occurring so that it can be replicated and (b) 
where it is lagging so that we can change it. Finally, the various agencies and consortia across 
the state must coordinate efforts to make certain that all districts in the state can access high- 
quality opportunities to learn about and be supported in the process of improvement. Given 
the current wide variation in access to local supports (e.g., Moffitt et al., 2018; Plank, 2018), the 
state must assure that all districts are given the resources to use data effectively to improve 
outcomes for their students. 
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Appendix: Real-Time Data Systems That Support Improvement 
Illuminate Education 


IIluminate Education is an example of a real-time online platform that encourages data 
sharing and integration (Blair, 2017; “Illuminate Special Education Information System,” n.d.). 
The application program interface (API) provides teachers and other stakeholders access to a 
range of data, such as student behavior, special needs, demographics, and formative test 
scores, that are updated in real time. The company offers four core products: Illuminate DnA 
(Data and Assessment), which houses a platform for teachers to create, administer, and analyze 
formative assessments to improve instruction; Illuminate K-12 Student Information system, 
which contains aggregated school data in a central location; Illuminate eduCLIMBER, which is a 
cloud-based interactive tool providing educators with easy data visualization; and Illuminate 
Special Education, which streamlines special education processes. One district that has been 
utilizing Illuminate DnA is Denver Public Schools (DPS) in Colorado, the largest district in the 
state (Getting Smart Staff, 2017). As of the 2016-2017 school year, Illuminate DnA has been 
implemented in 61 of the 142 schools within the district. DPS teachers are able to use 
IIluminate DnA to not only test their students after lessons but to also conduct more 
comprehensive interim and end of year assessments. 


PowerSchool 


Currently serving 32 million students in 13,000 school districts, PowerSchool is an online 
real-time platform that helps to reduce data silos within districts by housing student 
information in one location (“PowerSchool, a leading K-12 education technology platform,” 
n.d.). The platform’s PowerSchool Learning management system allows educators to customize 
and rearrange student data to best fit their needs as well as to share work with colleagues. For 
example, Oregon Community Unit School District utilizes PowerSchool’s central student 
information system to pinpoint struggling students without the cost of user machine 
installations (“Oregon CUSD 220 Improves Graduation Rate, Reduces Failing in Classes,” n.d.). 
The district, which previously had separate databases and Excel spreadsheets, relied on 
PowerSchool to identify that 30 percent of the freshmen class in 2008 received at least one F 
and to reduce the rate to 19 percent by 2015. PowerSchool is currently expanding, with an 
expected $1.8 million going into establishing itself in Roanoke, Virginia (Jones, 2017). 


Schoolnet 


Schoolnet, which is an online platform created by Pearson, provides a central system for 
educators to administer tests and see the real-time reports of the assessments next to the 
district’s student information system. The platform offers four modules: the Account module 
helps track student performance in accordance with school improvement plans; the Align 
module ensures that the district curriculum and instruction are aligned with student 
performance; the Assess module streamlines the process of administering benchmark 
assessments; and the Outreach module allows districts to share information with the greater 
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community. The highlighting feature of Schoolnet, however, is that district leaders are the 
“driving force” of the platform; that is, districts get to choose the specific services provided to 
their educators within the system. For example, Austin Independent School District (AISD) in 
Texas turned to Schoolnet to implement an integrated data warehouse that created 
“actionable” data for teachers and kept all information in one system (Schoolnet, 2012). 
Through the new system, teachers in AISD are given student profiles and behavior histories to 
which they did not access before. As a case study of three different districts using Schoolnet 
conducted by the Consortium for Policy Research in Education (CPRE) in 2004—2006 found, 
however, teachers who receive more training on using Schoolnet are more likely to be 
optimistic about its value, and district staff are key in successful implementation (Consortium 
for Policy Research in Education, 2007). 


Ed-Fi 


Ed-Fi is an open-source technology that consists of a uniform data standard, a 
centralized operational data store (ODS) and application program interface (API), and various 
mapping tools (“What is Ed-Fi?” n.d). Given the lack of consistency in the numerical codes 
assigned to students in various data systems that districts use, Ed-Fi created a uniform “Ed-Fi 
Data Standard” that allows districts using Ed-Fi to connect student information, assessment, 
and rostering systems. Tennessee’s Shelby County Schools, for example, has allowed educators 
to keep a roster of students who may need additional help in an after school tutoring program 
in one location rather than logging onto different systems (Kebede, 2016). Many independent 
platforms currently use Ed-Fi standards and technology, such as Certica’s Data Connect and 
InnovateEDU’s Cortex. The Cortex draws from the Ed-Fi ODS to collect data indicators multiple 
times a day and allows educators to view this information (Certica Solutions, 2017). Relatively 
new, InnovateEDU’s platform was recently piloted at Providence Public Schools in Rhode Island 
and Achieve First charter schools in the 2016-2017 school year (InnovateEDU Cortex, n.d.). 


Edupoint 


Edupoint’s Synergy Education Platform contains multiple systems, such as the Synergy 
Student Information System (SIS), Assessment, Learning Management System (LMS), Analytics, 
and Special Education (Synergy Analytics, n.d.). The SIS, for example, includes metrics such as 
demographics and discipline, dashboards, and over 300 pre-built reports so that teachers can 
analyze data in real time throughout the year. By condensing all of the various data 
management pieces into one central platform, Edupoint’s Synergy allows districts such as Mesa 
Public Schools in Arizona to input student information online so that educators do not have to 
manually update the data and parents can view the indicators as well (Edupoint Educational 
Systems, 2017). Synergy has also recently partnered with San Diego County Office of Education 
to implement its platform in several of the county’s school districts. 
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Schoolzilla 


Schoolzilla is a K-12 online data platform that integrates a data warehouse with school 
dashboard profiles and protected data sharing options (“Why Schooizilla?” n.d.). The platform, 
which updates district information on a daily basis, allows educators and administrators to view 
data points by subgroup as well as by student. Rocketship Education charter schools, for 
example, currently use Schoolzilla to serve their 90 percent free-and-reduced price lunch 
student population (Tansey, 2014). Although Rocketship Education teachers were previously 
unable to see detailed benchmark test data in order to inform instruction on a student level, 
they now have access to reports created through Schoolzilla’s platform that track an individual 
student’s performance on test subsections throughout the year. 


BrightBytes 


BrightBytes offers its Clarity and DataSense platforms that follow its unique approach: 1) 
integration of data; 2) research through partnerships with universities and organizations; 3) 
analysis of needs; and 4) data-driven recommendations. The K-12 Clarity platform provides 
data analytics support that converts complicated analyses into actionable next steps for 
improvement. The DataSense platform is BrightBytes’s data integration solution that collects, 
translates, and cleans data in a centralized location for customers. In addition, DataSense 
allows for a streamlined process of reporting data to the State Department of Education and 
makes it easier for educators to track and share data points from different sources. In 2017, 
BrightBytes partnered with San Bernardino County Superintendent of Schools to assist in 
establishing data transparency under the new San Bernardino Countywide Education Open 
Data Platform (BrightBytes, 2017). More recently, the platform has announced an eight-year 
collaboration with Georgia’s Henry County Board of Education to support the district’s 
personalized learning efforts through creating a one-to-one digital learning environment 
(BrightBytes, 2018). BrightBytes is also currently expanding its offerings from its award-winning 
Early Warning module and designing a Student Success module that will identify both students 
at-risk of not graduating and of not succeeding in postsecondary school. 


Student Insights (Somerville Public Schools) 


A few districts, such as Somerville Public Schools (SPS), have created their own open 
source data software in recent years (“The Boston Foundation awards Somerville Public Schools 
a two-year $200K grant,” 2017). SPS’s Student Insights software gives its schools access to real- 
time information on student performance, history of student behavior, and recommendations 
from teachers and administrators. The software also sends teachers email alerts if a particular 
student’s absences increase unusually. Through this centralized data system, SPS encourages its 
teachers to make incremental improvements to their teaching practices based on their students 
throughout the year. Although SPS is currently expanding Student Insight’s features to include 
analytics and an option for students to input information about themselves into the system, 
several other districts, such as Fall River and New Bedford, are already starting to implement 
Student Insights. 
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Project Unicorn, A Data Interoperability Initiative 


Alongside independent online platforms that are currently being created, there have 
been recent efforts to create communities around the idea of strengthening data sharing in the 
K-12 education system. Project Unicorn is one example of a collective alliance of over 200 
school systems, foundations, non-profit organizations, and vendors that advocates for secure 
interoperability in districts across the nation (Project Unicorn: A Data Interoperability Initiative, 
2017). Although the alliance does not recommend products or data standards, it provides a 
shared rubric for districts and vendors to assess whether they are moving towards improved 
data interoperability and enabling educators to make informed improvements in their 
classrooms. 
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